fix(slurm): detect srun inside interactive salloc via SLURM_STEP_ID#21712
Open
1fanwang wants to merge 1 commit into
Open
fix(slurm): detect srun inside interactive salloc via SLURM_STEP_ID#217121fanwang wants to merge 1 commit into
1fanwang wants to merge 1 commit into
Conversation
Inside an interactive `salloc` allocation, `SLURM_JOB_NAME` is `bash` or `interactive`, so `_is_srun_used()` returned `False` even when the script was actually launched as `srun python ...`. This emitted a spurious "srun is available but is not used" warning and made `SLURMEnvironment.detect()` skip a real SLURM job. `srun` sets `SLURM_STEP_ID` on the spawned process while `salloc` alone does not, so checking for that variable distinguishes the two cases. Fixes Lightning-AI#20776
1fanwang
requested review from
ethanwharris,
justusschock and
tchaton
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #20776.
_is_srun_used()returnedFalsewheneverSLURM_JOB_NAMEwasbashorinteractive. Those values are inherited from an outersallocallocation, but a nestedsrun python …inside that allocation IS a real srun invocation. The old check gave the wrong answer in that case, triggering a spurious "srun is available but is not used" warning and makingSLURMEnvironment.detect()skip a real SLURM job.The fix keys off
SLURM_STEP_IDfirst —srunsets it, baresallocdoes not. So ansalloc → srun python …workflow is now correctly classified as srun-used. The pre-existingbash/interactiveexclusion stays sosallocalone (no inner srun) still bails out cleanly.How was this PR tested?
Added two tests in
tests/tests_fabric/plugins/environments/test_slurm.py:test_no_srun_warning_inside_salloc_with_srun— setsSLURM_JOB_NAME=interactiveandSLURM_STEP_ID=42, asserts_is_srun_used()returns True and no spurious warning fires.test_detectto cover the same shape.Both fail without the change and pass with it.
Before submitting
src/lightning/fabric/CHANGELOG.md > Unreleased > Fixed)Documentation preview: https://pytorch-lightning--21712.org.readthedocs.build/en/21712/